Sensing images and light sources

ABSTRACT

The description relates to capturing or sensing color images of scenes and information about the type of light (e.g., light source) that illuminated the scene. One example can include an image sensor manifest as an array of pixels. This example can also include a lens configured to focus an image from a scene on the array of pixels. The array of pixels can entail first pixels that are configured to sense the image and second pixels that are configured to capture information about lighting of the scene.

BACKGROUND

Colors captured in a color image of a scene can depend upon the type oflight source illuminating the scene. Adjustments can be made to thecolors based upon the type of light source to make the color image morepleasing to a user.

SUMMARY

The description relates to capturing or sensing color images of scenesand information about the type of light (e.g., light source orillumination source) that illuminated the scene. One example can includean image sensor manifest as an array of pixels. This example can alsoinclude a lens configured to focus an image from a scene on the array ofpixels. The array of pixels can entail first pixels that are configuredto capture the image and second pixels that are configured to captureinformation about lighting of the scene (e.g., visible lightilluminating the scene).

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate implementations of the conceptsconveyed in the present document. Features of the illustratedimplementations can be more readily understood by reference to thefollowing description taken in conjunction with the accompanyingdrawings. Like reference numbers in the various drawings are usedwherever feasible to indicate like elements. Further, the left-mostnumeral of each reference number conveys the FIG. and associateddiscussion where the reference number is first introduced.

FIGS. 1 and 7 show example image and light source sensing systems inaccordance with some implementations of the present concepts.

FIGS. 2, 5, and 6 show example image and light source sensing processesin accordance with some implementations of the present concepts.

FIGS. 3-4 show example filters in accordance with some implementationsof the present concepts.

OVERVIEW

The description relates to cameras and more specifically to cameras thatcan detect information about the light illuminating a scene captured bythe camera. From another perspective, the concepts relate to capturingor sensing color images of scenes and information about the type oflight (e.g., light source) that illuminated the scene. Different typesof light, such as natural light, incandescent light, various fluorescentlights, light emitting diode (LED) lights, etc., can influence colors ofthe sensed color image. Thus, a scene illuminated with fluorescent lightmay appear different (in a sensed color image) than the same sceneilluminated with incandescent light even when captured by the samesensor. This is sometimes referred to as sensor metamerism. Knowing thetype of light source can allow various adjustments to be made to thesensed color image to generate a more pleasing image (e.g., coloraccurate or enhanced image) for a user. The present implementations cancapture information about the light source on the same image sensor thatcaptures the image with little or no perceivable degradation of thesensed or captured image. Stated another way, the presentimplementations can utilize the same image sensor to sense the image andto sense information about the light source by using different filters.Thus, the image sensor can be thought of as having extended lightdetecting or sensing capability without significant loss in sensed colorimage quality and without the addition of another sensor. Thus, a singleimage sensor can perform both functions while maintaining essentiallythe same sensed image quality.

FIG. 1 shows an example spectral sensing system 100. System 100 caninclude a camera 102 and a light source 104 that can illuminate a scene106. In this case, the scene is a flower in a vase. The camera caninclude a lens 108 and an image sensor 110. In the illustratedconfiguration, the camera 102 is a single-lens reflex (SLR) type camera.The present concepts are also applicable to other camera types. Camerascan capture two-dimensional or three dimensional color images. Notefurther, that as used herein the term ‘camera’ can refer to afreestanding device, such as the single-lens reflex (SLR) cameraillustrated in FIG. 1 or a viewfinder camera, and/or ‘camera’ can referto elements of a device (e.g., a smart phone device can include acamera). Examples of the latter scenario are illustrated and discussedrelative to FIG. 7.

In FIG. 1, the image sensor 110 can include an array of sensors 112 thatincludes multiple individual pixel sensors 112(1)-112(N) (fourteen pixelsensors are illustrated in this example). The image sensor 110 can be anactive pixel sensor, passive pixel sensor, or other type of sensor.Examples of active pixel sensors can include complementarymetal-oxide-semiconductor (CMOS) sensors or charge coupled devices(CCD), among others.

Filters 114 can be positioned over individual pixel sensors 112. Notethat the illustration is a simplified version of the pixel sensors 112and filters (e.g., filter array) 114. In practice, the number of pixelsin an image sensor 110 is often in the thousands or even millions, whichof course cannot be effectively illustrated. However, the concepts canbe effectively conveyed with this simplified version.

The filters 114 can include a first type of filter (e.g., first filtertype) 116 and a second type of filter (e.g., second filter type) 118.The first type of filter 116 can be configured to aid underlyingindividual sensors 112 to capture a portion (e.g. a pixel) of a sensedcolor image 120 of the scene 106. Stated another way, the first type offilter can be thought of as a color filter so that individual underlyingpixel sensors are sensitive to a specific human perceptible color, suchas red, green, or blue. From one perspective, the filter transmits acolor (e.g., transmits wavelengths of the color) and blocks otherwavelengths. Thus, the first type of filter allows the underlying pixelsensor to sense/capture color information that contributes to the sensedcolor image 120. As such, the first type of filters 116 can be thoughtof as a color filter array. Thus, in the illustrated configuration ofFIG. 1, the first type of filters 116 are color specific filters thatcontribute color information of the sensed color image 120. In thiscase, the specific colors are red (R), green (G) and blue (B). However,other implementations can use more and/or different color filters, suchas clear, cyan, magenta, yellow, and/or any other color system.

The second type of filter 118 is configured to cause the underlyingindividual sensors 112 to capture visible spectrum information about thelight source 104 rather than contributing to the sensed color image 120.The visible spectrum information about the light source 104 can bemanifest as scene light source metadata 122 associated with the sensedcolor image 120. As used herein the term ‘visible spectrum’ means thehuman visible/perceptible spectrum (e.g., a range of electromagneticradiation wavelengths from approximately/about 400 nanometers (nm) toapproximately/about 700 nm). The term ‘about’ is used since not allhumans are sensitive to exactly the same wavelengths and thus the 400 to700 range generally cited is an approximation of the general population.In this context, in one example, the term ‘about’ can mean +/−50 nm tocover all humans at both ends of the visible spectrum.

In some implementations, the second type of filter 118 can be thought ofas spectral band (SB) filters. Individual spectral band filters cansense a specific frequency/wavelength range of light that can provideinformation about the light source 104. The relative number of thesecond type of filters can be substantially less (e.g., exponentiallyless) than the number of the first type of filters 116. For instance, asensor with 400,000 pixels might have 399,000 filters of the first typeand 1,000 of the second type. The second type of filters can be randomlydistributed among the first type of filters (or otherwise distributed ina manner that does not have a significant visual impact on the sensedimage). In fact, relative to the sensed image, the pixels associatedwith the second type of filters can be treated as (or in a similarmanner to) defective pixels. For instance, a color value for the pixelsassociated with the second type of filters can be determined based atleast in part upon the colors of other proximate pixels, among othersolutions. The determined color value can then be assigned to the pixelrelative to the color image.

In some implementations, individual spectral band filters can be usedcooperatively to sense the entire visible light spectrum of 400 to 700nanometers. For instance, a first spectral band (SB1) filter could sense400 to 450, a second spectral band (SB2) filter could sense 450 to 500and so on with the last one (e.g., SBN) sensing 650 to 700.

The visible spectrum information from the various spectral band filterscould be combined to represent the visible light spectrum of the scenewhen a white light field is imaged. Other implementations can use fewerspectral band filters that are selected to sense specific frequencyranges associated with individual types of light sources. For instance,incandescent light sources tend to emit light having a generallycontinuous profile in the visible spectrum. In contrast, other lightsource types, such as fluorescent bulbs, tend to emit discontinuousbands of light. Specific fluorescent bulbs tend to have distinct peaksor bands at known wavelengths. Similarly, LEDs tend to emit light inpeaks that can be distinguished from the various fluorescent bulbs. Thesensed visible spectrum information from these sensors alone and/orcombined with the sensed color information from the sensors (underlyingthe first type of filter) can provide spectral information for theentire visible spectrum. In some implementations, this visible lightspectrum information and/or some facet thereof) can be saved as sensedlight source metadata 122. Other implementations may determine what typeof light the source is and save the source type (e.g., fluorescent) asthe sensed light source metadata 122.

FIG. 1 shows an implementation where each pixel of the sensor 110 haseither a filter of first filter type 116 or a filter of the secondfilter type 118. An alternative implementation can position filters ofthe first filter type over all of the pixels and then position thesecond filter types over some or all of the first filter type. Forinstance, a second filter layer could be positioned over the firstfilter types that is transparent (e.g., clear) for most pixels, butcontains the second filter type over some of the pixels such that thecombination of the first filter type and the second filter type producesa specific desired transmission range.

To summarize, some implementations can embed specialized pixels in thecolor filter array 114 to produce unique spectral sensitivity other thanthe RGB filter sensitivity without significantly impacting the sensorperformance.

FIG. 2 shows a device implemented process 200 for enhancing images. Theprocess can operate on the sensed light source metadata 122 associatedwith the sensed color image 120 to determine information about the lightsource, such as a type of light source (e.g., illuminant type) at 202.While this discussion mentions a single type of light source, thepresent implementations can detect multiple types of light sourcesilluminating the scene and a relative intensity of each light source(e.g., the relative influence of each light source on the scene). Otherimplementations may treat the sensed light source type as internalparameters in the image processing pipeline hardware and/or software.

The process can perform post image processing, such as color correctionon the sensed color image based upon the sensed light source informationat 204. In some implementations, the post image processing can employlight source specific color correction algorithms 206 that correspond tothe type of light source identified at 202. The light source specificcolor correction can adjust color values (e.g., intensity) from thesensed image to corresponding color values that would be expected if theimage had been illuminated with neutral light. From one perspective, thelight source specific color correction can be manifest as a light sourcespecific color correction algorithm that can adjust the color values ofindividual color pixels (e.g., R, G, B) from the sensed values toneutral light values based upon the type of scene lighting. As usedherein, ‘neutral light’ can be light having a pre-defined spectralprofile that is objectively or subjectively pleasing to human users. Thepredefined spectral profile could be defined by a device manufacturer ora component supplier. Alternatively, the predefined spectral profilecould be user selectable, such as from a set of neutral light optionsprovided on the device.

In the illustrated configuration, the light source specific algorithmscan include a natural light algorithm 206(1) (e.g., natural light toneutral light), an incandescent light algorithm 206(2) (e.g.,incandescent light to neutral light), a fluorescent light algorithm206(3) (e.g., fluorescent light to neutral light), and/or an LEDalgorithm 206(N) (e.g., LED light to neutral light), among others.

The process can produce an enhanced image 208 that compensates for theproperties of the light source and is more visually appealing to theuser. In contrast to previous auto white balancing techniques, thepresent process does not rely on estimating information about the lightsource from the sensed color image 120. Instead, the sensed light sourceinformation allows much more accurate and reliable auto white balancingthan the previous auto white balancing estimation techniques.

FIG. 3 shows a representative portion of filters 114 where the firsttype of filters 116 are red (R), green (G), and blue (B) generallyarranged in a Bayer arrangement (other arrangements can alternatively beused). FIG. 3 also shows a graph 300 of amplitude (e.g., intensity) andwavelength. The blue filters are configured to transmit light in theblue wavelengths (e.g., approximately 450-495 nm wavelengths). The greenfilters are configured to transmit light in the green wavelengths (e.g.,approximately 495-570 nm wavelengths). The red filters are configured totransmit light in the red wavelengths (e.g., approximately 620-750 nmwavelengths). In this implementation, the second type of filter 118includes SB1 and SB2 filters that are configured to detect fluorescentlight. The SB1 filter is configured to detect the 400-450 nm light andthe SB2 filter is configured to detect 600 to 650 nm. In practice,second type filters could be employed to detect other wavelength ranges(e.g., SB3, SB4, SBN). Further, while a single instance of the SB1filter is shown and a single instance of the SB2 filter is shown, manyimplementations can employ multiple instances of each filter (e.g.,multiple non-adjacent pixels having the same filter, such as a 600 to650 nm transmissive filter).

As illustrated, fluorescent light tends to have peaks at approximately425, 490, 540, and 620 nm. The SB1 and SB2 filters are configured todetect the 400-450 nm and 600 to 650 nm wavelengths, respectively. Thesewavelengths may not be readily detected through the red, green, or bluefilters. Thus, selecting ranges that include these wavelengths for theSB filters can provide useful spectral information for determining alight source that is illuminating the filters 114. Stated another way,the wavelength range defined by the second type of filter 118 can beselected to be valuable for identifying specific light source types.From still another perspective, individual second type filters can beuniquely more responsive to certain light sources, such as the spectralemission lines of fluorescent lamps. Thus, the addition of the secondtype of filters to filter array 114 can allow fluorescent lamps to bedetected and/or distinguished from other light sources, such asincandescent lights based upon specific wavelength spectral peaks.

FIG. 4 shows another representative portion of filters 114 that includesthree illustrated second type filters 118. For discussion purposes, FIG.3 also shows an instance of graph 300 without a sensed spectrum shown onthe graph. In this case, the second type filters SB1, SB2, and SB3 areconfigured to transmit light in wavelength ranges not captured by thefirst type filters R, G, and B. In the illustrated implementation, SB1can transmit 350-400 and SB2 can transmit 400-450. Ranges 450-570 arecaptured by the blue and green color sensors. The range from 570-620 canbe captured by second type filter SB3 and the range from 620-750 can becaptured by the red color sensor. Stated another way, a set of secondfilter types could be selected that have different spectral transmissionpeaks that are also different from the spectral transmission peaks ofthe first filter type 116 (e.g., red, green, and blue). In this way, bycombining information from the two filter types, the entire visiblespectrum associated with a sensed image can be captured. In theillustrated implementation, the SB filter ranges do not overlap with theranges of the color filters but are adjacent to the color filters. Forinstance, SB3 senses 570 to 620 nm and the red color sensor senses620-750. However, SB3 could for example, sense from 570 nm to 650 nm andpartially overlap the red color sensor. This could be valuable wheredeterminative peaks occur at the edge of the adjacent ranges. Forinstance, assume that the wavelength 620 nm is strongly indicative offluorescent lights. In such a case, having SB3 extend to 650 nm couldprovide effective detection of this wavelength and thereby provide moredeterminative results than having abutting ranges.

Still other implementations can dedicate second type filters 118 for theentire visible spectrum rather than relying upon the first type colorfilters 116 (and underlying sensors) for some of the spectruminformation. For instance, ten consecutive 50 nm bandwidths from 300 nmto 800 nm can be captured with ten different second type filters 118 togive some overlap at each end of the visible spectrum. Alternatively,six consecutive bandwidths of fifty nanometers each can be used to spanfrom 400-700 or five consecutive bandwidths of sixty nanometers each canbe used to span from 400-700, among other configurations.

Due to physical constraints on the drawing page only 48 pixel filtersare illustrated in a four by twelve array. In reality, the filters andunderlying pixels can number in the thousands or millions. In relationto the second type of filters 118, filters can be selected that arespecific to different wavelength ranges than the wavelength rangesillustrated here. For instance, a total of 100 filter pixels could bededicated to sensing the visible light spectrum. These 100 filter pixelscould be distributed, such as randomly distributed, over pixels of theimage sensor 110. The 100 filter pixels could be divided into ten groupsof ten, with each group dedicated to a particular wavelength range. Thepresence of these 100 pixels distributed among multiple thousands ofpixels sensing the color image allows the 100 pixels to be essentiallyinconsequential to the quality of the sensed color image.

FIG. 5 shows a device implemented process 500 that can employ some ofthe concepts described above. Block 502 can receive information frompixels of an image sensor responsive to the image sensor sensing a colorimage of a scene. In some implementations, the information can includesignals generated by the pixels in relation to sensing the image. Theinformation can also include information about the locations of thepixels on the sensor, the type (and/or subtype) of filter positionedover the individual pixels (e.g., what wavelength range of visible lightthe individual pixel/filter combination is configured to sense, etc.).

The sensed color image of the scene can be based upon a set of colors,such as red, green, blue, or cyan, yellow, magenta, or cyan, yellow,green, magenta, among others. In the image sensor, a majority of thepixels can be configured to detect one color of the set of colors (e.g.,a subset of the majority dedicated to red, another subset dedicated togreen, and a third subset dedicated to blue). A minority of the pixels(e.g., some or all of the remaining pixels) can be spectral band filtersthat transmit other portions of the visible spectrum to differentunderlying pixels that do not contribute color information to the sensedcolor image. Instead, these pixels can be used to capture other visiblespectrum information relating to illumination of the scene representedin the sensed color image.

Block 504 can identify other pixels (e.g., some or all of the remainingpixels described relative to block 502) that are not dedicated todetecting (e.g., sensing) one of the set of colors. In someconfigurations, the relative location of each pixel of the image sensorand the type (and sub type) of filter positioned over each pixel isknown. In such implementations, the locations of the other pixels (e.g.,the pixels that have the second filter type positioned over them) areknown by corresponding pixel locations on the sensor and are thereforereadily identified.

Block 506 can organize the other pixels into groups based uponwavelength ranges (e.g., portions of the spectrum) that individual ofthe other pixels are configured to sense. In some cases, the organizingthe other pixels can be organizing the output of the other pixels intogroups. For instance, the output (e.g. signals) of all of the otherpixels sensing 350-400 nm can be grouped and the output of the all ofthe other pixels sensing 400-450 nm can be grouped, and so forth. Insome cases, within an individual group the process can evaluate thesignals from the member pixels. The signals can be digital or analog. Insome cases, the signals can entail a profile (e.g., amplitude/intensityover a wavelength range).

In some implementations, the evaluating can identify pixels (e.g. pixelshaving outputs) that are outliers (or otherwise have reduced analyticvalue) and discard the outliers and evaluate remaining pixels of thegroup. For instance, the process may look for sharp peaks in the profilecompared to the adjacent wavelengths to identify bands that aredistinctive to (or indicative of) individual light sources, such asfluorescent lights. However, an individual pixel may be receiving lightfrom a portion of the scene that is highly colored in a similarwavelength light. In such a case, the peak may be ‘hidden’ (e.g., theprofile may have a high intensity proximate to the wavelength of thepeak, such that the peak is not pronounced when evaluating the signaleven though it (e.g., the peak) may be present). This pixel (e.g., theoutput of this pixel) may be of diminished diagnostic value compared toothers in its group and may be discarded. Of course, other evaluation ofthe visible spectrum information provided by the pixels can beperformed.

As mentioned, in some cases the evaluating can entail evaluating aspectral profile of a wavelength range of signals from individualremaining pixels of the group. In other cases, the evaluating can entailevaluating relative intensity of the signals from the individualremaining pixels at individual wavelengths.

Block 508 can utilize the information from the groups to identify a typeof light source illuminating the scene. In some cases the informationcan be utilized collectively to generate a spectral profile of visiblewavelengths illuminating the scene. The spectral profile can begenerated using solely the information from the other pixels (e.g., theinformation from the other pixels may be sufficient to recreate theentire visible spectrum of the image and/or may capture wavelengthranges that can definitively distinguish different light types.Alternatively, this information can be utilized to detect peaksrepresenting spectral banding without knowing the entire visiblespectrum. The spectral banding can be distinctive of specific types oflight sources. Stated another way, detected spectral bands at specificwavelengths can be indicative of a specific type of light source, suchas a fluorescent light.

Alternatively, the information from the other pixels can be utilized incombination with other information, such as signals from the colorpixels, among others. Such an example is shown in FIG. 4 where thevisible light spectrum is generated using signals from the other pixelsand the color pixels. The spectrum or parts thereof can be compared tospectral profiles of specific light sources. Still anotherimplementation of block 508 is described below relative to FIG. 6.

Block 510 can process the sensed color image based upon the type oflight source illuminating the scene to produce an enhanced image. Theaccurate light source identification offered by block 508 can allowvarious types of post image processing to be employed with lessresources and/or more accurate correction that existing techniques. Forinstance, auto white balancing algorithms specific for the identifiedlight source type can work more robustly and efficiently, which willresult in higher image quality and less camera response time. Oneexample of the processing is applying light source specific algorithmsto the sensed color image. Examples of light source specific algorithmsare illustrated and described relative to FIG. 2.

Block 512 can cause the enhanced image to be displayed for a user.Alternatively or additionally, the enhanced image can be stored forlater use. The enhanced image may be associated with the original (e.g.,raw) sensed color image, the spectral information, and/or the type ofidentified light source. For example, some or all of this informationcan be associated with the enhanced image and/or the raw image asmetadata.

FIG. 6 shows another device implemented process 600. This process can bethought of as an additional way of accomplishing block 508 of FIG. 5.Block 602 can obtain information relating to identified other pixels(e.g., the pixels underlying the second filter type 118) from block 506(of FIG. 5). In this implementation, block 604 can also identify grayregions of the sensed color image. Gray regions can be thought of asregions (e.g., groups of pixels) of the sensed color image that are grayor neutral colored.

Block 606 can identify whether any of the other pixels are in the grayregions. Stated another way, the gray region comprises a group of pixelsat a location and the process can identify whether the group of pixelsincludes any of the other pixels.

Block 608 can determine a ratio of a signal profile of an individualother pixel compared to signal profiles of individual color pixels inthe gray region. Stated another way, in an instance where one of theother pixels is included in the group of pixels, the process can comparea signal profile from the other pixel to signal profiles of the colorpixels (e.g., the process can compare the output of the other pixel whenexposed to the gray color to the output of the color sensors whenexposed to the gray color). Also, the process can access a data table ofstored ratios. These stored ratios can be generated under controlledconditions on an individual device, or globally, such as for anindividual device model as part of the product development. The storedratios can be obtained by capturing images of a gray surface incontrolled light environments (e.g., with natural light, withincandescent light, with fluorescent light, with LED light, etc.). Theratio in each circumstance can be stored in the data table and mapped tothe light source type.

Block 610 can compare the ratio from block 608 to the known ratiosproduced by known light sources to identify the type of light source.For instance, the ratio obtained at block 608 can be compared to thestored values to identify a potential match. For example, a similaritybetween the ratio and an individual stored ratio above a predefinedthreshold can be considered a match. The match can indicate the type oflight source. Among other uses, this information about the type of lightsource can be output to block 510 of FIG. 5.

FIG. 7 illustrates an example system 700 that shows various deviceimplementations for sensing images and light sources. In this case, fourdevice implementations are illustrated. Device 702(1) is manifest as asmart phone, device 702(2) is manifest as wearable smart device in theform of a smart watch, device 702(3) is manifest as a tablet, and device702(4) is manifest as a server in remote resources 704, such ascloud-based resources. Camera 102 of FIG. 1 is also another type ofdevice 702. Devices can communicate over one or more networks 706. Whilespecific device examples are illustrated for purposes of explanation,further examples of devices can include traditional computing devices,such as personal computers, cell phones, smart phones, personal digitalassistants, wearable devices, consumer devices, gaming/entertainmentconsoles, vehicles, or any of a myriad of ever-evolving or yet to bedeveloped types of devices.

Individual devices 702 can be manifest as one of two illustratedconfigurations 708(1) and 708(2), among others. Briefly, configuration708(1) represents an operating system centric configuration andconfiguration 708(2) represents a system on a chip configuration.Configuration 708(1) is organized into one or more applications 710,operating system 712, and hardware 714. Configuration 708(2) isorganized into shared resources 716, dedicated resources 718, and aninterface 720 there between.

In either configuration, the devices 702 can include a display 722,storage 724, a processor 726, a camera 728, a communication component730, and/or a light source identification (ID) component 732. Individualdevices can alternatively or additionally include other elements, suchas input/output devices, buses, graphics cards (e.g., graphicsprocessing units (CPUs)), etc., which are not illustrated or discussedhere for sake of brevity.

The camera 728 can include the lens 108, image sensor 110, andassociated filters 114 of FIG. 1. The communication component 730 canallow individual devices 702 to communicate with one another vianetworks 706. The communication component can include a receiver and atransmitter and/or other radio frequency circuitry for communicatingwith various technologies, such as cellular, Wi-Fi (IEEE 802.xx),Bluetooth, etc.

Light source identification component 732 can be configured to receiveoutput (information/data) from the image sensor 110 (FIG. 1). The lightsource identification component can identify which data is colorinformation of the sensed color image of a scene and which data relatesto spectral information of light that illuminated the scene. The lightsource determination component can then determine what type of lightsource illuminated the scene. Several processes that can be employed bythe light source identification component 732 are described aboverelative to FIGS. 5 and 6. The light source determination component canthen take actions to improve the sensed color image. An example processthat can be employed by the light source determination component isdescribed above relative to FIG. 2.

Note that not every instance of light source identification component732 needs to provide every functionality described above and/or performevery function in every circumstance. For purposes of comparison, firstconsider a robust implementation, such as might be accomplished on thecamera 102 of FIG. 1 or the smart phone device 702(1) of FIG. 7. In sucha case, the device can capture or sense a color image, identify the typeof light source based upon information from the second type filters 118(FIG. 1), produce an enhanced image 208 (FIG. 2) based upon the type oflight source, and display the enhanced image for the user. However, in asituation where the device's battery is low (e.g., below a definedthreshold), the light source identification component 732 may simplystore the sensed color image with the information from pixels associatedwith the second type filters 118 (FIG. 1). The light sourceidentification component 732 may then perform the other actions at asubsequent time, such as when the device is plugged in to a powersupply.

In an alternative configuration, the device 702 may be resourceconstrained and as such employ a less robust light source identificationcomponent 732 that may perform a limited functionality regardingidentifying light sources and/or enhancing images. For instance, thesmart watch implementation of device 702(2) may lack one or more ofprocessing, storage, and/or power resources. In this case, the lightsource identification component 732 can then store and/or transmit thesensed color image and associated information from the pixels of thesecond filter type 118 to another device, such as device 702(4). Thisdevice 702(4) can then perform the remaining functionality to producethe enhanced image(s) 208. The enhanced images can then be stored, suchas in the user's cloud storage, returned to the original device, and/orsent to another device.

From one perspective, any of devices 702 can be thought of as computers.The term “device,” “computer,” or “computing device” as used herein canmean any type of device that has some amount of processing capabilityand/or storage capability. Processing capability can be provided by oneor more processors that can execute data in the form ofcomputer-readable instructions to provide a functionality. Data, such ascomputer-readable instructions and/or user-related data, can be storedon storage, such as storage that can be internal or external to thecomputer. The storage can include any one or more of volatile ornon-volatile memory, hard drives, flash storage devices, and/or opticalstorage devices (e.g., CDs, DVDs etc.), remote storage (e.g.,cloud-based storage), among others. As used herein, the term“computer-readable media” can include signals. In contrast, the term“computer-readable storage media” excludes signals. Computer-readablestorage media includes “computer-readable storage devices.” Examples ofcomputer-readable storage devices include volatile storage media, suchas RAM, and non-volatile storage media, such as hard drives, opticaldiscs, and/or flash memory, among others.

As mentioned above, configuration 708(2) can be thought of as a systemon a chip (SOC) type design. In such a case, functionality provided bythe device can be integrated on a single SOC or multiple coupled SOCs.One or more processors can be configured to coordinate with sharedresources 716, such as memory, storage, etc., and/or one or morededicated resources 718, such as hardware blocks configured to performcertain specific functionality. Thus, the term “processor” as usedherein can also refer to central processing units (CPUs), graphicalprocessing units (CPUs), controllers, microcontrollers, processor cores,or other types of processing devices.

Generally, any of the functions described herein can be implementedusing software, firmware, hardware (e.g., fixed-logic circuitry), or acombination of these implementations. The term “component” as usedherein generally represents software, firmware, hardware, whole devicesor networks, or a combination thereof. In the case of a softwareimplementation, for instance, these may represent program code thatperforms specified tasks when executed on a processor (e.g., CPU orCPUs). The program code can be stored in one or more computer-readablememory devices, such as computer-readable storage media. The featuresand techniques of the component are platform-independent, meaning thatthey may be implemented on a variety of commercial computing platformshaving a variety of processing configurations.

ADDITIONAL EXAMPLES

Example implementations are described above. Additional examples aredescribed below. One example can include an image sensor comprising anarray of pixels. The array of pixels can include a filter arraypositioned over the array of pixels. The filter array can comprise afirst type of filter positioned over a majority of the pixels and asecond type of filter positioned over a minority of the pixels. Thefirst type of filter is configured to provide light filtering so thatspecific colors of human perceptible light are transmitted to underlyingpixels and captured as color information of a sensed color image of ascene. The second type of filter comprises spectral band filters thattransmit other portions of the visible spectrum to different underlyingpixels that do not contribute color information to the sensed colorimage but capture other visible spectrum information.

Another example includes any of the above and/or below examples wherethe spectral band filters are selected to capture bandwidths that areindicative of specific types of light sources or wherein the spectralband filters are selected for individual ranges of bandwidths that whentaken collectively cover an entirety of the visible spectrum from about400 nanometer (nm) wavelengths to about 700 nm wavelengths.

Another example includes any of the above and/or below examples wherethe colors of the first type of filter are associated with transmissionwavelengths that overlap transmission wavelengths of the second type offilter or wherein the transmission wavelengths of the first type offilter do not overlap transmission wavelengths of the second type offilter.

Another example includes any of the above and/or below examples furthercomprising a light source identification component configured to receiveinformation from the image sensor. The light source identificationcomponent is further configured to distinguish other visible spectruminformation from the color information of the received information.

Another example includes any of the above and/or below examples wherethe light source identification component is further configured toevaluate a profile of the other visible spectrum information to identifya light source of the scene.

Another example can receive information from pixels of an image sensorin response to the image sensor sensing a color image of a scene. Thesensed color image of the scene can be based at least upon a set ofcolors and wherein a majority of the pixels are configured to detect onecolor of the set of colors. The example can identify other pixels thatare not dedicated to detecting one of the set of colors. The example canorganize the other pixels into groups based at least upon wavelengthranges that individual of the other pixels are configured to sense. Theexample can utilize the information from the groups to identify a typeof light source illuminating the scene. The example can process thesensed color image based upon the type of light source illuminating thescene to produce an enhanced image. The example can cause the enhancedimage to be displayed for a user.

Another example can include means for receiving information from pixelsof an image sensor in response to the image sensor sensing a color imageof a scene. The sensed color image of the scene based at least upon aset of colors and wherein a majority of the pixels are configured todetect one color of the set of colors. The example can include means foridentifying other pixels that are not dedicated to detecting one of theset of colors. The example can include means for organizing the otherpixels into groups based at least upon wavelength ranges that individualof the other pixels are configured to sense. The example can includemeans for utilizing the information from the groups to identify a typeof light source illuminating the scene. The example can include meansfor processing the sensed color image based upon the type of lightsource illuminating the scene to produce an enhanced image. The examplecan include means for causing the enhanced image to be displayed for auser.

Another example includes any of the above and/or below examples wherethe receiving information comprises receiving signals output from thepixels.

Another example includes any of the above and/or below examples wherethe identifying comprises identifying the other pixels from informationfrom the image sensor about relative locations and filter typeinformation of individual pixels on the image sensor.

Another example includes any of the above and/or below examples wherethe organizing, further comprises within an individual group,identifying pixels that are outliers and discarding the outliers andevaluating remaining pixels of the group.

Another example includes any of the above and/or below examples wherethe evaluating comprises evaluating a spectral profile of a wavelengthrange of individual remaining pixels of the group, or wherein theevaluating comprises evaluating relative intensity of the informationfrom the individual remaining pixels at individual wavelengths.

Another example includes any of the above and/or below examples wherethe utilizing comprises utilizing the information collectively togenerate a spectral profile of visible wavelengths illuminating thescene.

Another example includes any of the above and/or below examples wherethe utilizing comprises utilizing the information to detect peaksrepresenting spectral banding.

Another example includes any of the above and/or below examples furthercomprising utilizing a presence of the peaks at specific wavelengths toidentity the light source.

Another example includes any of the above and/or below examples wherethe utilizing further comprises identifying gray regions of the sensedcolor image and identifying whether any of the other pixels are in thegray regions This example can further include determining a ratio of asignal profile of an individual other pixel compared to signal profilesof individual pixels in the gray region and comparing the ratio to knownratios produced by known light source types to identify the type oflight source.

Another example includes any of the above and/or below examples wherethe comparing comprises accessing the known ratios in a data table anddetermining a similarity between the ratio and individual known ratios.

Another example includes any of the above and/or below examples wherethe comparing identifies a match when the similarity exceeds apredefined threshold.

Another example can include an image sensor comprising an array ofpixels and a lens configured to focus an image from a scene on the arrayof pixels. The array of pixels can include first pixels that areconfigured to capture the image and second pixels that are configured tocapture information about visible light of the scene.

Another example includes any of the above and/or below examples wherethe image sensor comprises a charge-coupled device (CCD) or acomplementary metal-oxide semiconductor sensor (CMOS).

Another example includes any of the above and/or below examples manifestas a smart phone, a tablet, a wearable smart device, a single-lensreflex camera, a viewfinder camera, or a consumer device.

CONCLUSION

The described methods or processes can be performed by the systemsand/or devices described above, and/or by other devices and/or systems.The order in which the methods are described is not intended to beconstrued as a limitation, and any number of the described acts can becombined in any order to implement the method, or an alternate method.Furthermore, the method can be implemented in any suitable hardware,software, firmware, or combination thereof, such that a device canimplement the method. In one case, the method is stored oncomputer-readable storage media as a set of instructions such thatexecution by a processor of a computing device causes the computingdevice to perform the method.

Although techniques, methods, devices, systems, etc., pertaining toimaging and light source identification are described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described. Rather,the specific features and acts are disclosed as exemplary forms ofimplementing the claimed methods, devices, systems, etc.

1. A device, comprising: an image sensor comprising an array of pixels;a filter array positioned over the array of pixels; and, the filterarray comprising a first type of filter positioned over a majority ofthe pixels and a second type of filter positioned over a minority of thepixels, the first type of filter is configured to provide lightfiltering so that specific colors of human perceptible light from avisible spectrum are transmitted to underlying pixels and captured ascolor information of a sensed color image of a scene, and wherein thesecond type of filter comprises spectral band filters that transmitother portions of the visible spectrum to different underlying pixelsthat do not contribute color information to the sensed color image butcapture other visible spectrum information.
 2. The device of claim 1,wherein the spectral band filters are selected to capture bandwidthsthat are indicative of specific types of light sources.
 3. The device ofclaim 1, wherein the colors of the first type of filter are associatedwith transmission wavelengths that overlap transmission wavelengths ofthe second type of filter.
 4. The device of claim 1, wherein the colorsof the first type of filter are associated with transmission wavelengthsthat do not overlap transmission wavelengths of the second type offilter.
 5. The device of claim 1, further comprising a light sourceidentification component configured to receive information from theimage sensor, and wherein the light source identification component isfurther configured to distinguish other visible spectrum informationfrom the color information of the received information.
 6. The device ofclaim 5, wherein the light source identification component is furtherconfigured to evaluate a profile of the other visible spectruminformation to identify a light source of the scene.
 7. A deviceimplemented process, comprising: receiving information from pixels of animage sensor in response to the image sensor sensing a color image of ascene, the sensed color image of the scene based at least upon a set ofcolors and wherein a majority of the pixels are configured to detect onecolor of the set of colors; identifying other pixels that are notdedicated to detecting one of the set of colors; organizing the otherpixels into groups based at least upon wavelength ranges that individualof the other pixels are configured to sense; utilizing the informationfrom the groups to identify a type of light source illuminating thescene; processing the sensed color image based upon the type of lightsource illuminating the scene to produce an enhanced image; and, causingthe enhanced image to be displayed for a user.
 8. The device implementedprocess of claim 7, wherein the receiving information comprisesreceiving signals output from the pixels.
 9. The device implementedprocess of claim 7, wherein the identifying comprises identifying theother pixels from information from the image sensor about relativelocations and filter type information of individual pixels on the imagesensor.
 10. The device implemented process of claim 7, wherein theorganizing further comprises, within an individual group, identifyingpixels that are outliers, discarding the outliers, and evaluatingremaining pixels of the group.
 11. The device implemented process ofclaim 10, wherein the evaluating comprises evaluating relative intensityof the information from the individual remaining pixels at individualwavelengths.
 12. The device implemented process of claim 7, wherein theutilizing comprises utilizing the information collectively to generate aspectral profile of visible wavelengths illuminating the scene.
 13. Thedevice implemented process of claim 7, wherein the utilizing comprisesutilizing the information to detect peaks representing spectral banding.14. The device implemented process of claim 13, further comprisingutilizing a presence of the peaks at specific wavelengths to identitythe light source.
 15. The device implemented process of claim 7, whereinthe utilizing further comprises: identifying gray regions of the sensedcolor image; identifying whether any of the other pixels are in the grayregions; determining a ratio of a signal profile of an individual otherpixel compared to signal profiles of individual pixels in an individualgray region; comparing the ratio to known ratios produced by known lightsource types to identify the type of light source.
 16. The deviceimplemented process of claim 15, wherein the comparing comprisesaccessing the known ratios in a data table and determining a similaritybetween the ratio and individual known ratios.
 17. The deviceimplemented process of claim 16, wherein the comparing identifies amatch when the similarity exceeds a predefined threshold.
 18. A device,comprising: an image sensor comprising an array of pixels; a lensconfigured to focus an image from a scene on the array of pixels; and,the array of pixels comprising first pixels that are configured tocapture the image and second pixels that are configured to captureinformation about visible light of the scene.
 19. The device of claim18, wherein the image sensor comprises a charge-coupled device (CCD) ora complementary metal-oxide semiconductor sensor (CMOS).
 20. The deviceof claim 18, manifest as a smart phone, a tablet, a wearable smartdevice, a single-lens reflex camera, a viewfinder camera, or a consumerdevice.